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LATENT  FACTOR  MODELS  AND  ANALYSES 
FOR  OPERATOR  RESPONSE  TIMES 


D.  P.  Gaver 

Department  of  Operations  Rescareh 
Naval  Postgraduate  School 
Monterey,  CA  93943 


I.  G.  O’Muircheartaigh 
University  College 
Galway,  Ireland 


0.  INTRODUCTION,  BACKGROUND,  AND  SUMMARY 


There  are  many  situations  in  which  an  operator  (single  individual,  or*group 
or  crew)  is  confronted  with  a  somewhat  complex  task  that  must  be  accomplished 
within  prescribed  time  limits.  The  task  actually  often  initially  requires  diagnos¬ 
tic  steps  followed  by  action.  In  some  cases  the  diagnostic  steps  are  stimulated 
by  a  cue  event,  leading  to  probing  actions  intended  to  rev"!!  the  correctness 
of  a  tentative  diagnosis,  followed  by  observation  and  interpretation  of  system 
response,  in  turn  followed  by  viewpoint  revision  and  further  action.  While  it 
is  intriguing  to  attempt  to  model  response  in  such  detailed  terms,  this  paper 
does  not  embark  on  that  enterprise.  Rather,  we  provide  and  analyze  models  for 
the  resulting  overall  response  time  of  different  operators  to  different  tasks  where 
response  is  initiated  by  one  or  more  cues  provided  by  the  system.  Two  factor- 
analytic  models  are  presented  along  with  likelihood  estimation  procedures.  The 
latter  arc  then  employed  to  analyze  data  sets  from  typical  exercises  conducted 
at  simulators  used  for  training  nuclear  power  plant  operators;  their  identities 
are  kept  anonymous.  (The  findings  of  the  model  are  critiqued,  and  applications 
to  risk  analysis  are  sketched.) 
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It  is  believed  that  similar  models  will  be  useful  for  summarizing  the  behavior 
of  operators  or  crews  in  other  situations,  both  military  and  otherwise.  For 
example,  application  to  military  tank  driver  performance  is  envisioned. 
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1.  A  MIXED  OR  LATENT  FACTOR  MODEL  WITH  THE 

UNEQUAT,  CASE  FIXED  EFFECTS  AND  VARIANCES 

(LOG  N  MODEL). 

Consider  this  linear  model  of  mixed  type: 

^ik  =  ^ik  1=1,2,.../  ,  . 

k  =  1,2,.. .  K. 

where  Yik  =  InT,*;  wHh  T,^  being  the  time  for  crew  i  to  respond  to  situation  fc; 
ji  and  i/k  are  fixed  constants  (effects),  and  OJ,  ~  IIDN(0,  ct^),  6, it  ~  IIDN(0,  (t|), 
are,  respectively,  the  latent  rendom  component  that  “individualizes”  case  (in¬ 
dividual,  crew,  etc.)  i,  and  the  random  variation  displayed  by  any  individual 
on  situation  (task,  problem,  etc.)  k.  It  is  assumed  that  each  case  occurs  in 
conjunction  with  each  situation  (e.g.  a  person  confronts  a  particular  problem) 
just  once  in  the  data  set  to  be  modelled.  In  practical  circumstances,  some  such 
individual  interactions  may  be  missing  for  reasons  unrelated  to  individuals  and 
situations,  a  problem  that  is  deferred  for  the  present;  see  Appendix  A  and  B. 

As  implied,  the  model  described  may  well  be  of  interest  when  data  pertaining 
to  human  performance  are  to  be  analyzed,  but  should  also  be  of  use  elsewhere. 
The  A'  tasks  or  items  are  allowed  to  have  their  own  fixed  response  properties, 
described  by  (i^*;,rT^.);  this  pair  will  be  referred  to  as  a  task  signature.  The  usual 
mixed  ANOVA  model  formulation  assumes  (t^  —  a^,  constant  for  all  k  (see 
Scheffe  (1959)),  as  is  reasonable  when  measurement  error  is  represented. 

Note  that  because  of  the  assumption  of  possibly  unequal  ak's>  afixed-u;,  model 
cannot  be  usefully  estimated  by  likelihood.  Consequently  the  above  random- 
effect  model  has  been  introduced,  and  fitted  to  data.  As  will  be  apparent,  it 
is  possible  to  estimate  the  posterior  density  for  w,  using  Bayes’  formula  in  the 
style  of  empirical  Bayes;  the  mean  of  the  resulting  Normal/Gaussian  density 
/.’[u.',  I  data,  log  N  model],  is  available  as  an  estimate  of  the  ith  crew  effect  if  so 
desired. 

That  the  above  setup  is  a  latent  factor  model  has  been  remarked  to  us 
by  Professor  T.  VV.  Anderson;  see  Anderson  (1988)  Chapter  14  for  relevant 
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coverage  in  the  Normal/Caussian  case.  Brillinger  and  Preisler  (1983)  survey 
various  other  latent  factor  data- analytical  studies  in  non-Gaussian  settings;  this 
includes  a  detailed  discussion  of  a  latent  factor  Poisson  model  for  counting  data. 
Brillinger’s  paper  is  interesting  in  that  it  suggests  examining  goodness  of  model 
fit  by  “uniform  residuals,”  a  procedure  considered  in  our  study  as  well. 

Fitting  the  LOG  N  model  by  likelihood  requires  iterative  calculations;  the 
setup  is  described  in  the  next  section.  In  case  one  wishes  to  “robustify”  the 
formulation,  perhaps  by  introducing  more  outlier-prone  specifications  such  as  the 
Student  t  or  Tukey  density  for  a;,  then  more  numerical  effort,  or  approximation 
is  required.  Use  of  the  Laplace  approximation  together  with  Gauss-Hermite 
integration  may  well  turn  out  to  be  useful;  see  Gaver  and  O’Muircheartaigh 
(1987),  and  Gaver,  Jacobs  and  O’Muircheartaigh  (1990).  In  a  later  section  a 
totally  non-Normal/Gauss  model  for  operator  response  times  is  introduced  and 
fitted. 
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2.  FITTING  THE  LOG-NORMAL  MODEL  3Y  MAXIMUM 
LIKELIHOOD 

In  the  model  (1)  the  individualizing  ca^e  effect,  for  case  i  is  viewed  as 
a  latent  or  unobserved  rv  whose  effect  on  the  Yi^  observable  is  indirect.  What 
is  the  probability  distribution  of  Yik,k  —  1,2,. ..A"  in  terms  of  the  unknown 
parameters?  Clearly  it  is  multivariate  normal  since  u>,  and  occur  as  a  sum; 
the  density  for  case  (crew)  i  is,  by  conditional  independence,  given  , 


k=\ 


%/2xafc 


e  fc=i 

_  _ 

k=l 

To  obtain  the  unconditional  density  of  y.  remove  the  condition  on  u>,: 


e  2 


^2 


(3) 


The  calculation  needed  (“completion  of  the  square”  in  the  exponent)  can  be 
expeditiously  performed  as  follows.  Recognize  that  the  exponent  is  quadratic  in 
w;  put 


1  ^ 

'9  {y>k  /crl, 

k=\ 


(4) 


A',  being  independent  of  w.  To  find  and  A'i  differentiate  (2.3)  re  u>  and 

equate  coefficients  of  lj  and  1: 


(W,  —  U?)  ^  ^  /  2 

*=1 


so,  from  the  w-term, 


1/-^^  = 

k=\ 


(5) 

(6) 


Ft 


while  from  the  1-  term 
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and  hence  the  iog  likelihood,  /,  may  be  expressed  as 


2 =  /  Inr^ 


ln(r^  +  cT2)-^  {u,f  f -\- al) 

fc=i  «=i  «=i 

DifTerentiation  gives  these  estimates: 


(13) 


dl 


t 

=  0  =  [{y^k  -  y.-)  -  {I'k  -  I'-)]  , 


»=1 


so 


d{i‘k  -  U.) 

=  )  E  J'-t  -  )  E  E 


(14) 


i^k- 


(15) 


=  (y-k-y-)- 


Next, 


SO 


/i+j/.  =  y.. 


(16) 


(17) 


These  of  course  closely  resemble  conventional  ANOVA  estimates. 

When  estimating  variances  it  is  convenient  to  reparameterize  in  terms  of 
precision:  pk  =  1/(T^,  p  =  IIt'^  =  ELi  =  Ek=i  Pk- 

Then 

=  0 = -t  +  ^  -  4^  -  Ai(‘)  -  (m 


dpk 


p  Pk  1/p  +  (tJ 


(1/P  +  <^5) 


where 


and 


Next 


A](A:)  =  -  I'--)]" 


1=1 


1=1 


dcT?, 


r^  +  <Tl  {T^  +  alf' 


(19) 

(20) 

(21) 
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yiel  ds 

1  2  1  ^2  1  2 

-  +  =  7(^)'  =  7  E  ;  (‘^2) 

introduction  of  (21)  into  (18)  simplifies  the  latter  to 

-  =  -A](k)  +  -,  k=l,2,...,K.  (23) 

Ph  I  P 

The  system  ( 13),  ( 16),  (22)  and  (23)  must  be  solved  iteratively.  Begin  by  simply 
fitting  as  if  to  estimate  /r(  l),j/fc(l),Li;(l),  and  obtain 

1  ' 

JZi:  E  (24) 

from  which  compute  H7(2)  =  (1/^5(U)  /  (Hhzi  W^f)-  Next  calculate  2) 

y.^(2)  -  y..(2)  using  lT/t(2)  in  (15),  and  /i+t'(2)  =  y..(2)  from  (16).  ft  is  now 
possible  to  evaluate  A]{k\2)  from  (19),  and  (uj)^(2)  from  (20),  and  hence  pfc(2) 
and  p(2)  from  (23),  after  which  (^^(2)  from  (22).  Now  recompute  HT(3)  = 
(1/(t^(2))  /  -  Pk(2)/p{2),  and  so  repeat  iteratively  until  conver¬ 

gence  is  achieved.  A  solution  procedure  based  on  Newton-  Raphson  iteration 
has  also  been  obtained;  agreement  of  the  two  procedures  is  generally  good. 
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3.  LOG-EXTREME- VALUE  MODEL  (THE  LOG  EV  MODEL) 

An  altcrnativo  to  the  previous  model  that  may  be  attractive  is  the  following 
setup; 

(a)  7’,fc  is  distributed  according  to  a  two-parameter  Weibull;  then  it  follows 
mathematically  that 


(h)  Y,k  =  In  I'tic  has  the  extreme  value  distribution; 

1  —  exp  [  —  0,  exp  ({I/,fc  —  Tjk)  /^k)]-,  with  probability  density  function 


/v.k  {yik'-.'nk,^k\^i)  - 

exp[-0,  cxp((y,k  -  Ik)  /^k)]0.exp((ytk  -  rik)/^k)-^ 

Zk 


(25) 


Note  the  occurrence  of  parameter  ,  which  is  intended  to  represent  crew 
effect,  i.e.  0^  is  a  way  of  individualizing  crews  comparable  to  the  action  of  u,  in 
the  previous  model.  V'alues  of  6,  are  viewed  as  randomly  selected  latent  factors 
as  were  the  u,  values.  The  nature  of  the  0,  contribution  differs  from  Ui  in  this 
rnouel;  whereas  in  the  LOG  N  model  ui,  acted  purely  additively  (on  the  log  scale) 
to  affect  the  center  (mean  of  logged  response  times)  in  a  manner  common  to  all 
tasks,  in  the  f.OG  EV  model  it  can  be  seen  that  logged  times  are  represented  as 


y,k  =  +  +  (26) 

(,k  having  standardized  extreme  value  df.  For  the  present  model,  (25)  or  (26), 

E’  [V.fc l^.]  =  rjk  -  0.5772^^  -  In  (27) 

1.6449^1  (28) 

b 

which  permit  initial  parameter  estimation  by  moments  and  facilitates  compari¬ 
son  with  the  results  of  alternative  models.  Expression  (26)  implies  that  responses 
to  tasks  are  affected  differentially;  the  greater  the  natural  variation  in  perform¬ 
ing  a  task  by  crews  (measured  by  for  task  k)  the  greater  the  “average”  effect 
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on  task  duration  due  to  crew  effect.  This  is  a  specific  form  of  interaction  be¬ 
tween  crew  and  task  effects  that  may  (or  may  not)  be  reasonable  in  particular 
circumstances. 

Conditional  on  6i,  the  crew  I’s  response,  T,,  ^  different  tasks  has  joint 

density  function 

K 

(](,■’  0,),  (29) 

Jb=i 

where  co..ditional  independence  is  assumed.  In  order  to  obtain  the  unconditional 
joint  density  of  response  remove  the  condition  on  0i  by  integrating  out;  this 
step  corresponds  to  (10).  Thus 


fy,  {Vi',  {e 


(30) 


where 


and 


c,  =  J2exp({y,k  -  Vk)/^k) 
k=l 


d,  =  exp  (y.k  -  rik)/^k)  n 

\k=i  /  k=\ 


(31) 

(32) 


The  above  model  closely  resembles  one  introduced  by  Crowder  (1985)  and 
Crowder  and  Kimber  (1989).  However,  ours  deals  with  the  log  time,  and  hence 
is  a  location-scale  model  that  more  closely  compares  to  the  additive  log-normal 
model,  although  the  is  not  generally  a  mean,  nor  is  a-  standard  deviation. 


4.  GAMMA  VARIATION  FOR  9 

A  search  for  mathematical  tractability  suggests  that  variation  in  6  be  de¬ 
scribed  by  a  gamma  density: 


rii/0)  ft 


(33) 
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so  E[0]  =  1  and  Var[0]  =  /J,  and  from  which  the  joint  density  of  observations 
by  crew  i  is 

„  a  =  r(A-  +  i/in  (3,) 

r(i//3)  \i  +  pcj 

The  likelihood  associated  with  I  independent  crews  is 

or 

/  =  In  L  =  /A'ln/3  +  /ln(r(A'  +  l/i3)/r(l//?)) 


-(A'  +  l//3)^ln(l  +  /?c.)  +  ^lnd.. 

i=i  t=i 

After  arrangement  and  re-parameterization  so  that  4>k  =  In^jt  the  log-likelihood 
becomes 

^■-1  ^  ^  ^  ^ 

/  =  /^  ln(it  -I-  1/0)  -  ( A'  +  l/0)'^H^  +  0c,)  +  Yi  Yiy>>^  -  Vk)eM-<f>k)  - 

fc-n  i=l  <=1  k=\  k=l 


11 


5.  FITTING  THE  LOG-EXTREME  VALUE  (LOG-EV)  MODEL 
BY  MAXIMUM  LIKELIHOOD 


To  obtain  the  maximum  likelihood  estimates  of  the  parameters  we  iteratively 
solve  the  following  equations,  for  which  k=  I,2,...,A'  throughout: 

dl/df}  =  0, 

dl/dT]k  =  0,  (38) 

dlld4>k  =  0- 


One  Newton-Raphson  iteration  only  is  applied  to  each  equation,  after  which 
the  entire  process  is  repeated  until  convergence.  Typically,  only  two  or  three 
repetitions  are  required. 

We  record  the  derivatives  needed  for  the  above  process. 


dl 

dH 


-{K  +  1/(3)  ^  (c./(l  +  (3ci))  +  (1//32)  X;  ln(l  +  (3c,) 


1=1 


1=1 


K— 1 

^  l//3(l  +  fci3)  (39) 

k=0 


where  r^jt  =  {y,k  -  ■nk)/^k,  a  residual;  finally 


dl 

d<t>k 


=  (A'/J+  1) 


E 


/ 


r,ke^'<^l(  \  -3- 0Ci)-  '^Tik  - 
1=1 


/. 


(41) 
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The  second  derivatives  are 


02/ 

W 


/ 

=  /A7/32  +  (A-  +  1//3)  ^  c2/(l  +  daf  +  (2/02)  ^  c,/(l  +  0c.) 

i=l  i=l 

-(2/0^)  ^  ln(l  +  0c,)  -  (I/d*)  +  1/0)- 


i=l 


A'-l 


\-2 


-(2//0")  ^  (fc  +  l/0)-‘. 


fc=0 


(42) 


0  =  (1  +  A'0)c-2^*^  {(c''*V(l  +  /?c.))[(0c’-«V(l  +  /3c.))-l],  (43) 
"ic  1=1 

^  =  (1  +  A'0)  ^  |0  (rifce^'*/(l  +  0c, ))^  -  e'’'*^r,*.(l  +  r,it)/(l  +  0c, )| 

/ 

+  r.fc.  (44) 

1=1 

In  order  to  use  the  inverse  of  the  Fisher  information  matrix  to  provide  stan¬ 
dard  errors  of  the  parameter  estimates  all  cross-partial  derivatives  are  required; 
we  omit  recording  these  in  the  interest  of  brevity;  the  expressions  may  be  ob¬ 
tained  from  the  authors.  Our  numerical  experience  has  been  that  standard 
errors  obtained  from  Fisher  information  tend  to  be  too  small,  as  judged  from 
bootstrapping  approaches  next  to  be  described. 


6.  BOOTSTRAPPING 


A  modern  alternative  for  obtaining  standard  errors  and  approximate  confi¬ 
dence  limits  is  the  parametric  bootstrap  of  Efron  (1979;  esp.  Remark  K,  p.25). 
This  procedure  has  recently  been  applied  to  failure  data  in  the  context  of  the 
Challenger  disaster  by  Dalai,  Fowlkes  and  Hoadley  (1988)  and  goes  as  follows: 
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putfl  =  {4)l,,n,Ui,U2,  . .  ,a\)\i\  Model  LOG  N,and(/3;  t]^,  . . .  ,t}k\ 

in  Model  LOG  EV.  Note  that  6  =  (6i,. . .  ,Bp)  here  denotes  a  generic  parameter; 
it  bears  no  direct  relation  to  the  crew  effect  in  our  LOG  EV  model,  (26). 
Then  our  procedure  is  this: 

(a)  Estimate  6  from  data;  the  result  is  £(0),  the  point  estimate  of  the  param¬ 
eters. 

(b)  Provisionally  adopt  d(0)  as  the  true  value  in  the  parametric  model,  in  the 
present  case  (1)  or  (25). 

(c)  Simulate  B  independent  data  sets  (bootstrap  samples)  from  the  model 
evaluated  at  1(0):  {yijt(h),  i  —  1, . . . ,  /,  k  =  1,2, . . . ,  A';  6  =  1,2,...,  B}. 

(d)  Compute  estimates  of  d  for  each  sample,  obtaining  the  bootstrap  estimates 
{S.{b),b  =  1,2, . . . ,  fl}  =  {l(fl)},  the  bootstrap  distribution  of  1. 

(e)  Present  relevant  statistical  summaries  of  marginal  and  joint  distributions 
of  {l(fl)}:  e.g.  use  as  standard  error  of  1(0)  components  the  corresponding 
standard  deviations  of  the  bootstrap  estimate;  use  as  confidence  limits 
upper  and  lower  percent  points  of  the  bootstrap  sampling  distributions, 
suitably  adjusted.  We  present  numerical  illustrations  in  the  next  section. 

(f)  The  same  procedure  can  evaluate  standard  errors  of,  and  confidence  limits 
for  predictions  from  data:  in  the  present  case  prediction  of  the  probability 
that  a  re.sponsc  time  exceeds  any  given  value  is  evaluated  in  terms  of  the 
model  evaluated  repeatedly  at  bootstrap  parameter  estimate  values;  see 
balal  et  nl  (1988  )  for  an  example.  See  Section  9  for  an  example  in  the 
present  context. 

Bootstrapping  methods  suggest  themselves  for  comparing  the  adequacies  of 
different  models  for  fitting  and  predicting  from  specific  data  sets.  Specifically, 
bootstrapping  may  assist  in  choosing  between  two,  or  more,  candidate  models. 


■,4>k) 


14 


In  the  present  setting  one  may  wish  to  predict  the  probability  of  non-success, 
i.e.,  of  response  time  exceeding  some  time  window  of  duration  t,  P{t;  6).  Models 
A  and  B  (e.g.  our  LOG  N  and  LOG  EV  options)  are  estimated  obtaining  ^^(0) 
and  0^(0).  Then  generate  bootstrap  samples  for  A  and  B  using  flyi(O)  and 
^^(0)  respectively,  resampling  to  estimate  the  mean-squared  error  of  prediction 
when  Model  i  is  used  to  predict,  gven  that  the  data  comes  from  Model  j;  here 
=  {/I,  A;  A,  B;  B,B\  B,A}.  Prefer  the  model  whose  use  minimizes  the 
maximum  estimated  mean-squared  error  of  prediction.  An  alternative  strategy 
is  to  prefer  prediction  from  the  most  conservative  model:  the  one  predicting  the 
greatest  risk;  see  Section  9  and  Draper,  Hodges,  et  al  (1987). 

Another  option  for  residual  examination  is  to  compute  estimates  of  the  ex¬ 
pected  log  response  times  associated  with  each  Task/Crew  combination.  Since 
crew  effects  are  random,  we  estimate  them  in  specific  cases  by  their  posterior 
means. 

For  the  LOG  N  model,  examination  of  (10)  reveals  that  the  posterior  density 
cf  is  N  l(ui/r^)  /  (l/r^  +  .  W  (1/^^  +  substitute  in  the 

mle’s  for  the  various  parameters  to  estimate  in  a  particular  case: 

W.  =  (w./(T-)^)  /(l/r^T  l/ff^) 
and  then  from  (1),  (8),  (15),  (16) 

Vik  =  (m+i^,)  +  (t'fc-'^.)  =  y..  +  {y,k  -  y..)+  [(y,.  -  y..)/(ff]  /  [1/(t^)  +  l/^l] 

(45) 

where  all  averages  are  suitably  weighted.  Note  that  the  above  formula  for  the 
mean  acts,  in  effect  as  if  a  preliminary  hypothesis  test  for  homogeneity  of  crews 
is  being  applied:  if  is  very  small,  giving  evidence  that  all  crews  are  the  same, 
then  the  estimate  y,*  —  y.t,  the  (weighted)  task  mean  for  each  crew.  On  the 
other  hand  if  is  very  large  then  y,k  y.k  +  (yi.  -  y.,)>  the  task  mean  modified 
by  the  estimated  effect  for  crew  i.  The  effectiveness  of  such  a  smooth  transition 
when  pooling  data  was  noted  by  Mosteller  (1947). 
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For  the  LOG  EV  model  take  the  expectation  of  (27)  with  respect  to  the 
(estimated)  posterior  density  of  6i,  which  is  Gamma(c,  +  1//3,  A'  +  1//?)  from 
(33)  and  (34).  Using  the  first  two  terms  of  the  asymptotic  expansion  we  find 


Vik 


=  %  -  0.57724  -  4  [in  (A'  +  \/p)  ~  In  (c.  +  1/p)  ~  0.5/  (a^  +  1/p) 
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7.  EXAMPLE  DATA  ANALYSES 


The  previous  models  have  been  used  to  analyze  response  time  data  from  sim¬ 
ulator  experiments  involving  operators  performing  certain  safety-related  tasks. 
In  Tables  1  and  2  appear  actual  data  from  two  such;  System  L  and  System  D. 
It  is  noted  that  certain  task-crew  combinations  are  missing,  some  because  of 
simulator  failure.  Two  procedures  were  adopted  for  dealing  with  these  cases: 
(1)  values  were  imputed  by  an  EM-like  process;  see  Little  and  Rubin  (1987); 
alternatively,  (2)  a  likelihood  approach  was  taken  that  simply  omits  such  values 
from  the  analysis  by  setting  to  unity  the  likelihood  contribution  associated  with 
a  cell  having  a  missing  response.  Both  approaches  can  be  useful;  the  former 
leans  more  heavily  on  model  correctness.  The  analyses  reported  here  emphasize 
the  use  of  a  simple  imputation  procedure;  the  incomplete  data  results  are  also 
reported  in  the  summary  tables. 

In  addition  to  “true”  missing  values  there  are  observations,  here  marked 
with  asterisks,  that  were  judged  to  result  from  operators  following  non-standard 
response  strategies.  It  was  judged  to  be  useful  to  analyze  the  data  both  with, 
and  without,  including  such  values;  when  omitted,  those  nonentries  in  the  data 
table  were  treated  as  missing  values,  entries  imputed  as  above  or  treated  as 
missing  values,  and  analyses  made  using  LOG  N  and  LOG  EV  models. 

Results  of  fitting,  along  with  bootstrapped  standard  errors,  appear  in  Tables 
3  and  4  for  System  L,  and  Tables  5  and  6  for  System  D.  Both  tables  exhibit 
main  effects  (fi  -|-  i^k,Vk)  and  scale  parameters  (<7jtand^fc)  for  LOG  N/EV  mod¬ 
els  computed  under  A:  missing  values,  and  non-standard  strategy  values,  both 
treated  as  literally  missing,  using  methods  of  the  Appendix,  and  alternatively 
with  entries  imputed,  and  B:  only  the  “true”  missing  values  treated  as  missing. 
The  imputation  process  used  wcis  iteration  based  on  standard  two-way  ANOVA 
with  fixed  task  and  crew  effects.  This  is  a  convenient  crude  approximation  to  a 
proper  EM  algorithmic  approach.  Little  and  Rubin  (1987).  In  addition,  random 
crew  effect  variance  parameters,  and  respectively  for  the  two  models,  were 
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estimated.  Tiie  resampling-refitting  parametric  bootstrap  supplied  the  standard 
errors;  see  Efron  (1979)  and  Dalai,  Fowlkes,  and  Hoadley  (1988). 

It  is  noted  that  for  System  L  the  fixed  effects  under  A  and  B  agree 

closely,  with  the  exception  of  Tasks  2  and  9.  Data  for  Task  9,  in  Table  1,  exhibits 
8  out  of  18  missing  values,  and  a  further  4  non-standard  strategy  values,  all  of 
which  are  far  in  excess  of  other  times  for  that  task.  Data  for  Task  2  also  exhibit 
substantially  many  missing  values  and  non-standard  times,  the  latter  having 
resulted  from  operators  following  non-standard  procedures  and  hence  yielding 
times  more  lengthy  than  the  other,  acceptable,  values.  The  noticeable  differences 
are,  however,  still  within  2  bootstrap  standard  errors.  The  corresponding  scale 
effects  (e.g.,  log  task  time  standard  deviations)  for  Tasks  2  and  9  behave  in 
corresponding  fashion,  increasing  by  factors  of  2  to  3  if  the  non-standard  times 
are  included. 

Similar  behavior  occurs  for  System  D,  although  here  the  exceptions  occur 
for  Tasks  4  and  8.  There  are  fewer  missing  and  non-standard  times  reported 
for  System  D  than  for  System  L.  Relatively  large  changes  occur  in  the  standard 
errors,  as  well  as  in  main  effect  and  scale  parameter  estimates,  when  several 
missing  or  non-standard  times  are  encountered,  and  these  are  treated  differently 
in  the  analysis. 

In  order  to  check  for  the  effect  of  imputation,  and  also  for  that  of  apparent 
correlations  between  certain  task  times  the  analyses  were  re-run  for  System  L 
omitting  Tasks  2,  4,  and  9.  The  results  appear  in  Tables  7  and  8.  Although 
specific  numerical  values  are  changed,  the  general  pattern  remains  quite  similar: 
the  new  numbers  are  quite  often  well  within  a  standard  error  of  the  estimates 
that  utilize  data  from  all  tasks. 
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8.  MODEL  CRITICISM  VIA  RESIDUAL  EXAMINATION 


In  order  to  examine  the  overall  fit  of  the  models  to  data  it  is  useful  to  con¬ 
duct  some  form  of  residual  analysis.  We  have  chosen  first  to  judge  the  overall 
degree  of  fit  by  computing  and  summarizing  uniform  residuals  as  described  by 
Brillinger  and  Preisler  (1983).  In  general  for  this  procedure  one  estimates  model 
parameters  ff,  from  data  and  then  examines  the  estimated  probability  integral 
transformation  of  the  data,  utilizing  the  fitted  model:  if  the  model  is  correct 
then  the  latter  should  closely  resemble  the  uniform  distribution.  In  Figures  1 
through  8  we  display  plots  and  summary  statistics  for  such  estimated  proba¬ 
bility  integral  transforms  of  the  present  data  set.  We  also  exhibit  the  result 
of  bootstrapping  once:  each  model  was  allowed  to  create  one  set  of  bootstrap 
sample  data  utilizing  the  fitted  parameter  values;  these  values  were  then  treated 
like  raw  data,  and  were  then  probability  integral  transformed  and  the  results 
plotted  and  summarized. 

The  results  have  different  implications  for  the  appropriateness  of  the  models 
for  the  two  data  sets.  The  left  uniform  plot  of  Figure  1  shows  decided  non¬ 
uniformity  of  residuals  when  the  raw  data  is  fitted  by  LOG  N  using  the  methods 
of  the  Appendix;  however,  if  a  bootstrap  sample  is  generated  using  the  fitted 
model  parameters  the  results  are  far  more  uniform.  This  strongly  suggests  that 
the  basic  model  is  inappropriate.  A  similar  implication  is  obtained  by  examining 
Figure  3,  the  residuals  of  which  are  associated  with  LOG  EV.  Figures  2  and  4 
describe  the  residuals  when  imputation  of  missing  and/or  non-standard  values 
is  conducted.  Notice  that  uniformity  of  residuals  of  the  fit  of  the  raw  (plus 
imputed)  data  is  greatly  enhanced.  This  is  not  surprising  since  imputation  is 
based  on  presumption  of  model  correctness,  and  the  missing  and  non-standard 
values  are  imputed  u.sing  the  presumed  model.  The  same  general  behavior  is 
observed  when  data  from  System  D  are  fitted  by  the  two  models  in  various  ways. 
Here,  however,  the  departure  of  the  residuals  from  uniformity,  as  shown  on  the 
left  most  residual  plots  of  Figures  5  and  7,  seems  less  pronounced  than  is  the 
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case  for  System  L.  Again,  imputation  improves  the  unifonnity  of  the  residuals 
and  the  apparent  fit.  We  conclude  that  the  log-additive  models  are  more  likely 
to  be  trustworthy  for  system  D  than  for  System  L. 
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9.  MODIFICATION  OF  THE  MODELS  FOR  INITIAL  DELAY 


In  the  nuclear  plant  simulator  exercises,  and  doubtless  for  other  applications 
as  well,  certain  task  response  times  may  begin  after  a  cue  different  from,  and 
later  than,  the  actual  initiating  event.  That  is,  the  operational  cue  that  triggers 
response  may  occur  some  time  after  a  possible  original  cueing  event  (e.g.,  the 
first  evidence  of  nuclear  plant  abnormality).  Unfortunately,  the  time  of  the  op¬ 
erational  cue  is  not  usually  recorded  in  simulator  practice,  and  so  response  time 
data,  which  may  use  initiating  event  time  or  some  other  well-specified  event 
time  for  reference,  tends  to  exhibit  an  initial  delay.  Such  delays  may  be  inferred 
from  plant  and  initiating  event  information,  or  by  examining  the  actual  response 
time  data.  Note  that  ignoring  such  delays  if  they  are  appreciable,  i.e.,  fitting 
a  2-parameter  LOG  N  or  LOG-EV  model  when  a  3-parameter  specification  is 
more  appropriate  can  importantly  change  the  estimated  parameter  values,  par¬ 
ticularly  the  LOG  N  cr^  or  LOG  EV  (k,  the  measures  of  within-task  variability. 
Specifically,  if  the  delay  is  ignored  and  our  current  models  employed  uncritically 
when  a  delay  >  0  is  required  then  the  estimated  values,  d*  and  will  be 
biased  downwards  (under-estimated),  sometimes  quite  significantly. 

To  illustrate  the  effect  of  a  rough  accounting  for  delay  examine  the  Task  10 
data  for  system  L  in  Table  1.  The  minimum  value  is  1402  and  the  maximum 
is  3450,  so  there  is  an  appreciable  delay  associated  with  be^nning  the  task 
as  compared  to  the  variability  of  the  response  times.  If  the  data  is  taken  at 
face  value  and  our  LOG  N  and  LOG  EV  models  fitted,  then  Table  3  exhibits 
^10  =  0.28  considerably  smaller  than  that  for  other  tasks.  If,  however,  a  rough 
adjustment  is  made  for  delay  by  subtracting  1000  from  each  Task  10  response 
time  data  value  then  the  simple  standard  deviation  estimate  for  logged  (times- 
1000)  for  Task  10  becomes  0.59,  far  more  similar  to  other  task  response  time 
standard  deviations. 

At  pre.sent  the  LOG  N  and  LOG  EV  programs  fit  2-parameter  models,  so 
accounting  for  delay,  7jt,  must  be  done  off-line.  A  formal  approach  to  the  es- 
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tiniation  problem  for  the  Weibull  model  is  given  by  Smith  and  Naylor  (1987); 
that  paper  also  references  other  relevant  articles. 
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10.  RISK  CALCULATIONS 

An  important  application  of  the  LOG  N  and  LOG  EV  models  is  to  risk 
analysis:  it  is  desired  to  estimate  the  probability  that  a  task’s  response  time 
occurs  within  a  particular  time  window,  for  in  that  case  (some  aspect  of)  the 
threat  has  been  averted.  We  will  refer  to  the  probability  that  response  time 
exceeds  the  time  window  as  the  human  interaction  risk  associated  with  the  task. 

It  is  easy  to  make  point  estimates  of  the  required  probabilities  using  both 
existing  models:  one  simply  replaces  the  model  parameters  associated  with  the 
task  of  interest  by  their  maximum  likelihood  estimates.  A  natural  way  in  which 
to  handle  crew  effect  is  simply  to  remove  the  crew  condition  or  by  integrating 
it  out  with  respect  to  the  appropriate  NormaJ/Gauss  or  Gamma  estimated  prior. 
In  order  to  as.se.ss  the  effect  of  a  particular  crew  on  the  risk  it  is  necessary  to 
calculate  the  posterior  density  for  that  crew  and  then  integrate  out  on  w  or  ^  with 
respect  to  that  posterior.  It  may  well  be  of  interest  to  compare  the  estimated 
risk  as  it  depends  upon  which  crew  is  in  place  when  an  initiating  event  occurs 
in  order  to  a.ssess  the  effect  of  individual  crews  directly  on  risk.  This  is  not  done 
here. 
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11.  RISK  UNCERTAINTY 

In  order  to  assess  the  the  uncertainty  inherent  in  the  risk  estimation  one  may 
once  again  bootstrap.  The  procedure  is  this: 

(a)  Estimate  6  from  data  to  obtain  ^o- 

LOG  N  :  do  = 

LOG  EV: 

(b)  Provisionally  adopt  do  as  the  true  value  in  the  parametric  model. 

(c)  Simulate  B  independent  data  sets  from  the  model  with  parameter  do  : 
{y,k{b),  with  b  =  1,2,..., fl}. 

(d)  Estimate  d  from  each  sample  of  (c).  Obtain  {d(6),  6  =  1,2, ...,  i?}; 

(e)  Compute  >  wjfe|d(6)}  =  P{ln  T.k  >  In  ii;jt|d(6)}; 

=  pjEfc  >  w[\0{b)^  =  rk{b),  where 


Wf.  =  In  Wfc . 

The  risk  associated  with  Task  k  estimated  from  bootstrap  sample  6(6  = 
1,2, . . . ,  5)  for  each  model,  that  is,  calculate  the  probability  that  the  time 
window  Wk  is  exceeded  using  the  b‘^  set  of  parameters  estimated.  For  our 
two  models  and  6  =  1,2, .. .  ,B),  and  also  6  =  0,  the  original  estimate,  th’i : 
becomes,  respectively,  for  the  two  models, 

fij(6;LOG  N)  =  I (47) 

«fc(6;LOG  EV)=  (l+/l(6)exp(u;;-^fc(6))/a6))  (48) 
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The  following  table  exhibits  risk  calculation  for  prescribed  windows;  the  lengths 
chosen  are  for  illustration  only.  Calculations  have  been  made  both  without  non¬ 
standard  strategy  values  (A)  and  including  non-standard  strategy  values  (B); 
any  missing  or  left-out  values  were  imputed  as  before. 

Examination  of  the  point  estimates  in  Table  shows  that  there  is  considerable 
similarity  in  the  orders  of  magnitude  of  the  risks  given  by  LOG  N  and  LOG  EV. 
However,  LOG  EV  values  are  consistently  below  those  for  LOG  N,  as  are  the 
corresponding  confidence  limits.  It  is,  thus,  more  conservative  to  adopt  LOG  N 
model-based  risk  estimates  than  to  use  those  based  on  LOG  EV,  or  eqi  ‘valently 
the  Weibull  model.  The  exception.  Task  9,  is  probably  traceable  to  the  many 
missing  values  exhibited. 

(f)  Present  statistical  summaries  of  marginal  and  joint  distribution  of  Rk{b): 
e.g.,  use  as  standard  error  for  the  original  risk  estimate,  /ii(0),  the  stan¬ 
dard  deviation 


SRk 


\B-l 


(49) 


Note  that  the  overall  risk  associated  with  a  particular  initiating  event  de¬ 
pends  upon  the  risks  associated  with  all  tasks  (human  interactions)  associated 
with  response  to  the  event.  These  risks  may  well  be  dependent  probabilities, 
at  minimum  because  all  tasks  presumably  confront  the  same  crew.  To  handle 
the  dependency  induced  by  a  common  crew  one  can  assume  conditional  inde¬ 
pendence,  multiply  risks  conditional  on  crew  to  obtain  the  risks  associated  with 
joint  events,  and  then  integrate  out  with  respect  to  the  crew’s  posterior  density. 
The  calculation  may  be  performed  explicitly  for  the  LOG  EV  model  since  a 
closed-form  elementary  function  expression  for  the  extreme- value  survivor  func¬ 
tion  exists;  no  such  simple  calculation  can  be  carried  out  for  the  LOG  N,  but 
numerical  procedures  are  always  available. 


25 


Examination  of  the  results  suggests  considerable  similarity  between  the  risks 
calculated  using  the  LOG  N  and  LOG  EV  models.  It  is,  however,  noticeable 
that  for  all  tasks  except  Tasks  3  and  9,  LOG  N  predicts  a  slightly  greater  risk 
than  does  LOG  EV,  and  generally  with  slightly  larger  standard  error.  Of  course 
in  most  cases  shown  the  risks  associated  with  the  window  values  in  our  example 
are  too  high  to  be  realistic;  the  windows  were  chosen  for  illustration  only. 

The  above  calculations  have  been  carried  out  using  parameter  values  ob¬ 
tained  under  imputation.  Thus  the  apparent  similarity  of  risks  across  models  is 
probably  overstated,  and,  in  view  of  the  suspicion  cast  on  the  fits  to  system  L 
data  (see  Section  8)  we  should  treat  the  System  L  risks  with  caution. 
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12.  SUMMARY  AND  CONCLUSIONS 


In  this  article  we  have  suggested  and  shown  how  to  fit,  by  maximum  likeli¬ 
hood,  two  models  for  operator  response  times.  The  fits  of  the  models  to  two  sets 
of  (actual)  data  are  displayed  and  compared.  Uncertainties  are  assessed  by  para¬ 
metric  bootstrapping.  Complications  involving  missing  values  and  consequent 
lack  of  balance  are  dealt  with  by  direct  likelihood  computation  as  well  as  by  a 
simple  form  of  imputation.  Finally,  the  fitted  models  are  applied  to  estimate 
the  risk  of  exceeding  (hypothetical)  time  windows;  associated  uncertainties,  i.e., 
standard  errors  and  confidence  limits,  are  obtained  by  bootstrapping. 

We  view  this  work  as  a  pilot  or  feasibility  study  intended  to  illustrate  and 
explore  possibly  useful  approaches  and  methodology  to  an  important  area.  Out¬ 
standing  problems  remain:  the  models  put  forward  were  chosen  for  their  abilities 
to  account  for  some  aspects  of  the  real  situation  and  for  relative  tractability,  but 
many  other  forms  could  be  conjured  up,  fitted,  and  applied  to  infer  risk,  as  de¬ 
fined  here.  It  is  somewhat  interesting  to  find  that  risks  estimated  from  the  same 
data  using  the  different  models  agree  rather  closely;  it  is  not  unlikely  that  the 
agreement  will  suffer  if  the  windows  are  increased  so  as  to  achieve  much  smaller, 
and  presumably  more  realistic  risk  values. 

The  bootstrap  standard  errors  and  confidence  limits  warn  that  although 
the  present  models  tend  to  agree  in  their  risk  assessments  the  uncertainty  is 
still  rather  large,  even  if  wrong-model  or  structural  uncertainty  is  ignored;  see 
Draper,  Hodges  et  al  (1987)  for  discussion.  In  order  to  reduce  the  uncertainty  of 
estimation,  e.g.  to  reduce  standard  error  size,  it  is  often  proposed  to  aggregate  or 
pool  data,  either  from  similar  tasks  in  the  same  environment  (plant),  or  for  the 
same  task  across  “similar”  environments.  Both  procedures  are  worthy  of  investi¬ 
gation,  but  will  be  credible  only  if  suitable  adjustments  are  made  to  reduce  bias. 
Adjustments  can  be  carried  out  by  using  models  resembling  the  types  suggested 
here,  possibly  enhanced  to  include  regression  terms  “explaining”  responses  in 
terms  of  measured  and  qualitative  crew  and  plant  characteristics  (“performance 
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shaping  factors”  is  the  jargon  in  certain  risk  assessment  circles).  To  date,  for¬ 
mal  adjustment  attempts  by  regression  have  been  inconclusive  but  are  a  form  of 
insurance  that  should  be  included,  and  validated  to  the  greatest  extent  possible, 
if  aggregation  or  pooling  is  contemplated,  particularly  across  plants.  Inter- plant 
variability  may  be  appreciable  because  of  variations  in  management  philosophy 
and  style. 

In  general,  it  seems  advisable  to  utilize  bootstrapping  as  extensively  as  possi¬ 
ble  to  build  an  appreciation  for  the  variabilities  and  uncertainties  involved  when 
using  models.  Bootstrapping  that  uses  direct  re-sampling  as  originally  discussed, 
Efron  (1979),  seems  difficult  or  impossible  for  situations  such  as  are  described 
here  unless  vastly  more  data  becomes  available  and  better,  more  scientifically 
based,  models  and  true  replications  can  be  employed.  Consequently,  use  of 
parametric  bootstrapping  becomes  necessary.  However,  parametric  bootstrap¬ 
ping  that  consumes  data  from  more  realistic,  and  elaborate,  models  and  their 
fits  to  simpler  structures  can  be  useful  and  informative.  But  such  exercises  can¬ 
not  directly  substitute  for  data  obtained  under  truly  operational  circumstances, 
which  even  the  best  simulator  data  can  not  aspire  to  be. 
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APPENDIX  A.  MAXIMUM  LIKELIHOOD  ESTIMATION  FOR  LOG 
N  MODEL,  WITH  MISSING  VALUES 

Let  dik  =  1  if  the  observation  at  (i,k)  [i.e.,  for  crew  i,  task  k]  is  available; 
otherwise  =  0.  Then  the  component  of  likelihood  associated  with  (Crew)  i 
is 


K 

“5  H  /(jJ 

e  *=• 


n 


K 

k=l 


Now  write 


A  1  /  _  v2 

dik  {Vik  -  fi-  i^k-<^)  •  ~T  = 

it=l 

After  differentiation  re  u, 


which  implies,  identifying  terms  of  order  1  and  u>,  and  writing 

f  K 

E  d,k  iVik  -fi-  •'k)Pk  =  Wi/r? 

,  k=i 

E  d,kPk  =  1/T-i* 

k=\ 


Hence  ^  If  d,kPk 
k=\ 

K  K 

'^i  =  rf  Yi  dik  (Vik  -  P  -  t'k)Pk  =  Vi.  -  p  -  Y,  diknPk,  where  y,. 
k=\  *=! 

and 

K 

h'i  =  d^k  {Vxk-  p-Vk-^i  f  Pk  ■ 

k=:\ 


-  ,-2 


K 


'^i  E  d^kVikPk 

k=l 
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Thus  re-write  the  likelihood  as 


f  /  2\  -kw  [  ^  *^^**^' 

=e  ^  •  /  — .  .^d.rT~  ^ - 

OC  - 7-r - N - - 


{ n  4'^] 

[k=i  j 

Hence  the  likelihood  assumes  the  form 

^  -  llTTir 


-1  ULx 

Taking  logarithms,  we  obtain 


nA? 
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Differentiating  first  re  /r,  we  obtain 


dp 
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Vi.  -  M  -  TV*  E  dikt^kPk 


fc=i 


Next,  differentiate  re  uf. 
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Further  simplification  results  in 
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Hence 
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We  now  differentiate  re  pi,  noting 
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Noting  also  that 
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it  follows  that 
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di 
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Finally,  differentiating  re  <t^,  we  find 
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which  implies 
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1 


I 


fet  (r? + -’i)' 

The  following  iterative  procedure  was  employed: 


(a)  Use  imputed  values  to  obtain  A(0))^w(0)»7Jfe  (0),p*  (0)[fc  -  1,...,  A'] 

(b)  Solve  for  /i(l)  using  (A-1)  Update u>, 

(c)  Solve  for  j>/(1)  using  (A-2)  Update  u>i 

(d)  Solve  for  pi  using  (A-3)  Update 
(c)  Solve  for  a^  using  (A-4) 


Repeat  (b)  through  (e)  until  convergence. 


34 


REFERENCES 


Anderson,  T.  W.  (1984)  An  Introduction  to  Multivariate  Statistical  Anal¬ 
ysis  (Second  Edition).  New  York;  John  Wiley. 

Brillinger,  D.  R.,  Preisler,  H.  K.  (1983)  “Maximum  likelihood  estimation 
in  a  latent  variable  problem.”  Studies  in  Econometrics,  Time  Series  and 
Multivariate  Statistics.  New  York:  Academic  Press. 

Crowder,  M.  (1985)  “A  distributional  model  for  repeated  failure  time  mea¬ 
surements.”  Journal  of  the  Royal  Statistical  Society,  Ser.  B,  47,  pp.  447- 
452. 

Crowder,  M.  (1989).  “A  multivariate  distribution  with  WeibuU  connec¬ 
tions.”  Journal  of  the  Royal  Statistical  Society,  Ser.  B,  51,  No.  1,  pp. 
93-107. 

Dalai,  S.  R.,  Fowlkes,  E.  B.  and  Hoadley,  B.  (1990)  “Risk  Analysis  of 
the  Space  Shuttle:  Pre-Challenger  prediction  of  failure.”  Journal  of  the 
American  Statistical  Association,  84,  No.  408,  pp.  945-967. 

Denby,  L.,  and  Mallows,  C.  (1988)  “Robust  analysis  of  mixed  models.” 
AT&T  Statistical  Research  Reports.  No.  65.  AT&T  Bell  Laboratories, 
Murray  Hill,  NJ. 

Draper,  D.,  Hodges,  J.  S.,  Learner,  E.,  Morris,  C.  N.,  Rubin,  D.  B.  (1987). 
A  Research  Agenda  for  Assessment  and  Propagation  of  Model  Uncertainty. 
Rand  Note  N-2683-RC.  The  Rand  Corporation,  Santa  Monica,  CA. 

Efron,  B.  (1979)  “Bootstrap  methods:  another  look  at  the  jackknife.” 
Annals  of  Statistics,  7,  pp.  1-26. 

Gaver,  D.  P.,  and  O ’Mui rehear taigh,  I.  G.  (1987).  “Robust  empirical  Bayes 
analyses  of  event  rates.”  Technometrics,  29,  No.  1,  pp.  1-15. 


35 


Gaver,  D.  P.,  Jacobs,  P.  A.,  and  O’Muircheartaigh,  I.  G.  (1990)  “Regres¬ 
sion  analysis  of  hierarchical  Poisson-like  event  rate  data:  superpopulation 
model  effect  on  predictions.”  (to  appear  in  Communications  on  Statistics). 

Hougaard,  P.  (1986)  “Survival  models  for  heterogeneous  populations  de¬ 
rived  from  stable  distributions.”  Biometrika,  73,  pp.  387-396. 

Hougaard,  P.  (1986)  “A  class  of  multivariate  failure  time  distributions.” 
Biometrika,  73,  pp.  671-678. 

Kimber,  A.  C.,  and  Crowder,  M.  J.  (1988)  A  Repeated  Measurements  Model 
with  Applications  in  Psychology,  University  of  Surrey  Technical  Report  on 
Statistics,  No.  65. 

Little,  R.  J.  A.,  and  Rubin,  D.  B  (1987)  Statistical  Analysis  with  Missing 
Data.  New  York:  John  Wiley  and  Sons. 

Mosteller,  F.  W.  (1947)  “On  pooling  data.”  Journal  of  the  American 
Statistical  .Association,  43,  No.  242,  pp.  231-242. 

Scheffe,  H.  (1959)  The  Analysis  of  Variance.  New  York:  John  Wiley  and 
Sons. 

Smith,  R.  L.,  and  Naylor,  J.  C.  (1987)  “Statistics  of  the  three- parameter 
Weibull  distributions.”  Annals  of  Operations  Research,  9,  pp.  577-587. 

Watson,  A.  S.  and  Smith,  R.  L.  (1985)  “An  examination  of  statistical 
theories  for  fibrous  materials  in  the  light  of  experimental  data.”  Journal 
of  Materials  Science,  20,  pp.  3260-3270. 


36 


TABLE  1 

SYSTEM  L  TASK-CREW  TIMES 


TABLE  2. 

SYSTEM  D  TASK-CREW  TIMES 


TABLE  5 

SYSTEM  L  “  Missing  non-standard#  excluded 


^oot^JO^fooo^ooeo► 

OOOO'Of-^CMOOrHC 

q:  . 

LU  ooooooooooc 

O  'OOrOOfv.r-«>--ts70^f- 
_»  »0  O  CM  O  m  »0  O  NO  ♦« 

OOONOOO<-tOOOC 

ooooooooooc 


>0  O  NT  C 

^JO  o  rN.  f 

O  O  O  OnC 

Qt  .... 

lU  o  o  o  o  c 

2  wwwwv 

o  fN.  o  «o  r 

^  cnj  o  o  <-«  r 

o  o  O  c 


ONO.-«rv\oONtotOO»ONom 

cvjorN.ONj-coO'r'iorjofO 

rHOOO^<Sjm>-<ONTi-«»0 

Q£  . 

iJj  o  o  o  «-<  o  o  o  o  o  o  o  o 

Q_  •j‘m>o*-«<sio(Ma'NOO>ONr 
3  OOsO^Oc^O'^Or«.OC\l03^J■ 

rHOOooo.^rgoo«>^ocNi 

oooooooooooo 


r«-orNio»orN.osyor>*r 

<MOf-40oeo>-<rv.osT'C 

Cjo#>^o(si^ONOi^oin^ 

OOOpHOOOOOOC 

NOCNjcco<sj.-«»orN.oi-tr 

tno^fo^ooo^«^>.OM}^ 

(uo#>io(Mto«T.>Ho»a‘»’ 

ooo>-hooooooc 


wocoin(\»rN.«jooNONy»o 

»oo.-«ro»omcN»oor^cjNO 

oooooooooooo 

OOOQOOOOOOOO 

c^<MlnoNo.~^^0'T^oor^o 

»HO*-<rsCviNy<NJ«-»o<M.-«v;f 

oooooooooooo 
OOOOOOOOOOC}  o 


N3’ON3’«0(SJ»0'J0NOrsJ<>^{0 
♦  nO  o  »0  O  m  «0  On  NT  O  On  NT  On 

CX  OOOOOOOOOOOO 

UJ  . 

OOOOOOOOOOOO 

O  sT  (Sj  nA  CO  ^Nl  NO  ON  «-«  HO  nT 

fv.  o  NT  rsi  NO  ON  o  io  o  csj  m  ON 

to  O  O  O  O  O  O  •-«  O  O  f-4  o  o 

oooooooooooo 


oomNOO"Toooo-Horj 

COONTNOCOCONTtN-OONNONT 

ooooNo*-<NToo<srocNi 

oooooooooooo 

^HOON^^fnoNCNjofsjO'U^ 

sooHOONOfOtomooNTr*. 

OOOrN.O<>-«rHOOrHO.-H 

oooooooooooo 


OO^J^OC^^.eOC^ONTNO^J 

(SIONTONOr.NTM>OOrveO 

f~40ooN^rs)NTooNTorsi 

^  OOOOOOOOOOOO 

tn  WW  W  N^  W  W  W  W  N-#  W  N-» 

►-I  ooo.*-<‘0*ooc>JrN.or%fN.o 
oc  (Nj  o  irt  CO  (Nj  NT  1-4  NO  o  r>.  CO  h> 

r-^OOC^r-^fSI►0oOr^lO^i 

oooooooooooo 


OOOOOOOOOOOO 

eooNooocsjCNjooooo 
NO  ^  •~4  Cnj  rH  o  o  Cnj  CO  CO 


oooooooooooo 

«OOnOOO(NJNOOOOO 
•^OOfNjCOOO 
.-4  p-t  rsj  •-* 


OOOOOOOOOOOO 


OOOOOOOOOOOO 


ooocaoooooooo 


OOOOOOOOOOOO 


oooooooooooo 


OOOOOOOOOOOO 


^  OOOOOOOOOOOO 

O  ro.  qn  o  CO  csj  o  (Nj  r^.  o  ^ 

e0'O'O'ONO»O'T»nr^CNj0Nir» 

OOOOOOOOOOOO 


OOOOOOOOOOOO 


OOOOOOOOOOOO 


OOOOOOOOOOOCO 


OOOOOOOOOOOO 


mHONTiTiNTCsiTNO^or^iTir'. 


»orom»ANTCNj'Tmc>jr-^inr-^ 


>  •o»nNTm'TCMNTvocor«-Nnr>«. 
+  o  f-#  «n  nt  ni  ON  fNj  vT  nt  f  4 « 

^  NT  r- O' O' m  NO  in  p-4  nt  tn  NT 

>o»o»om'TfUNTNOHorN.mr'. 


HfjfOsTiO'or^cooNOr-irj 


.‘HrNjHONTtnNor-eco'O.-^Csi 


Figures  not  in  parentheses  are  estimates  when  no  imputation  is 
out  for  cases  with  missing  values  -  methods  described  in  the  Ai 
are  employed;  figures  in  parentheses  are  the  corresponding  es 
when  values  are  imputed  for  missing  cases. 


■D 

4) 


> 
\T  -4 
C 

iU  o 


u 

o 


tn 

<c 


bJ 

Cn 

O 


^OCNJtONTOv»Orsj»-«r^NT»0 

C3OO00OC3»0OO»-tO»"< 

oooaoooQOOOo 

►n\TCNirsj<r‘^cO'3‘Ofsj'3‘r-* 

•yOO(\lc^^orv^^■^JrJco^Ofsi 

ooomoo^C300o.-i 

OO0OO0OOC3OOO 


vjcM'or^.O'sa'r^Oirkf'HU 
(M»-<t0O»^00b^C0  00»O»- 
^4aoa»-^^Jsooovr»- 


2»- 

u*- 


^ooooocaoo 


CL  *-»  a  <M  O' O' tv  rH  IT*  hO  O' O' 
«^ooooo>~«cvioorHorNi 

OOOOOOOOOC3C30 


'^K*i,-imo'iO»oocM*^'0»o 

•  vj- o  fs» 'J' fM  NO  a  (S/ <M  r- fM 

OC  o  C3  O  O  O  Q  ^  O  O  O  O  O 

bi  . 

oooooooooooo 

a  o  <v»  o  \r  vr>  ^  in  o  o  in  s£> 

V—  f^C3^lO»-HfO(MrH^r-t»-HsT 
OOOOOOOOQOOO 

OOOOOQOOOOOC3 

>  <SJ 

ocor  • 

UJUJO 

vy  »/\ m in  o -H NO  cj  vT O'  ►-S'"' 
♦n o  Ny  in 00  oo «o  o -o  o  1''- 'X  ZO'O 

.-H  O  O  O' O  -X  O  C3  »o  <3  <\» 

^  ooooocaoooooo  >tj  o 

I.J 

>-l  <M  >0 o  O  »'S  nX  «n  sT  <Ni  O'  z 

o:  o  o  .-H  nx  <\i  eo  ^n  rn  o  in  fs-  uj 

^  o  c?  fN.  o '-4  •‘H  o  o  o  •><  o 

. . 

o  o  o  o  o  <d  o  o  o  o  o  o  u.  no 
Z  •'O 

oc^  ' 
<jujo 
a.  w 

2  . .  a.  O' 

O  a  o  o  o  o «  o o  o  o  C3  o  oO'X 

Cl  to  O  nO  o  O  N  <M  o  O  O  O  O  O' 

z  r-lOOCM00«0  o 


ft: 


oooooooooooo 


OOC3000000000 


O  C3  O  CO  O  O  O  O /-<  o  o  o 

ir>  o  in  (vj  m  cr  00  o  r'.  m  .-t  nX  uj 

•oo>Nor'.mrNJx*nooK>iO'X 

o  to  o  o  o  a  o  o  o  c»  o  o 


O  f'J  vT  'O  00  OJ  r 
CM  (NJ  CM  »-« «-i  *-i  • 


I  '-o  nT  O  -O  nX 
ir^<roN>-4 


oocoooooooooo 


vocoino'inoNDr'''^ONO^^'0 


OOOOOOOOr-COCOO 


►o\XXm\j'fMXNO'Tr-~inr-. 


ro  in  \x  CM  X  *n  »o  p' m  h- 


■»  CM  fo  X  m  o  C'-  00  O'  o  .•'«  cj 


oc 

lu 

3 


^  O  >0  O  O' 
X  O  OC".  *0  CM 
OGOOnO^^H 


in  -.o 

r-4  O 

►o  o 


in  o  r~t  o 
o  X  tM»n 
0(MO<-( 


o  o  o  o  o  o 

.-lOtncoCMO' 
m  o  o  O'  CM  to 
o  oo  «o  Cl  o 


o  o 
(Mto 


•-H  CM  X  O 
•-H  60 

000>-4 


ooooOooooooo 


mrv»-toQr''.»nin 

(Mf-^csjocMooinx 

»no^oCM'n'OrH 


a. 

3 


CA 


3 

O 

O 


NO  O  O'  N 

o  X  m  >0 
#-*  »n  1-^  X 


ooo<-toooooooo 


0^0'<MO•-^^00'NO 

K>fO'0O'O00Nor'> 

»Ao*-«oCsi»nx»~* 


X  .-INOX 
m  o  (O'M 
CMlA'MX 


OOOt-iOOOOCOOOO 


fMNovootn<MNO>n»ncMx'o 

ooo*n«“»in6oO'X»no'Xo 

oooooooocdociri 

0OOO0C3O0OO0O 

mxO'»-tONXtncMf-^inr'weo 

cO'HX'Ovoo'O'^r'^CMma' 

000000<'HOOf-«00 

oooooooooooco 


<  60 
>  -t 
ftrck:  • 

UJUiO 

<MX'0*-<oinfMN»A'0'Xin 

»-3w 

tooxO'(Mr'«0'r'*xo'«ooN 

zo  W 

rMOOO'''^NXOOlAO(M 

*~4^0 

oooooooooooo 

lu  o 

V) 

u 

73  0 

»n60^n>*0O'(MN0'0r»h>o 

z 

0  U  U) 

*ooineocM»n<-(soor''-oors. 

UJ 

•H-rl  9 

•-•OOOM-tCMiOOi-iCMOfM 

Ck 

i.7>+' 

►M 

U  C  A 

oooooooooooo 

u.  oo 

A  0  E 

Z  tn 

U  ft'rt 

Oft  ' 

av 

(JUJO 

Vl<  VI 

a.''' 

•  H  0 

^cftco 

0 

oooooooooooo 

oox 

CX  9 

CCOnOOOCMCMOOOOO 

O' 

O-H  C 

•^r-4f-4C\t  rMOOCMeOCO 

o 

•H  'H 

f~4  CM  rH 

■K  C73 
A  -H  C 

v  o 

oooocoooooooo 

NoiA--‘CMCMO'coi-<*nr''600 
rH  •— t  o  o  hm  in  o  ^  ^ 

oooocoooooecoo 


OOOOOOOO'MOOo 


> 

+ 

3. 


.-tXO'OtoCMOCMin'^'O-M 

O'tT"O'0'oinxinr"~tMO'ir> 


OOO0C3OOO>^C>OO 


Ki60NO»nxxcMX-Meo— 'in 
oooooooooooo 


OOOC0OOC>OOC9OO 


OOO'-iO'XOOOOOOCMO' 

n0(MO0'‘0'0C''.C0P'.>0X*-'» 

»oxx‘nxrMX'OXc>-mp. 


3 

D 


•nx»n»nx<MX'OXr'.«nr'. 


'-icgKix'0'0>'-«»0'0'-'f'j 


3*0  a 
a  «  M 
E  V 

•rl'H  1- 

L  C 
O  U  O 
C  Vt  u 

« 

CTJ  « 

«  jc 
X  «*H 
i-o 

O  •  M 
IftX  I-  V 
V-*'  <9  m 
’^9)  n 
to  E  «  U 
E  « 

•H  I  n  o 
» c 

VI  <AX  'H 
•  Ol-H  M 
3  C  V) 
4)<'M  0)>H 
C  (0  C  E 

o  >  «0 
a  c 

lA  01  O 
«»  C  Cf* 
ViH-H 
«  (A  'O 

r  VI  V)  o 

+'  rt 
c:  E  C  3 
0  3  a 

CX  Ol  E 

A-f'-H-H 

a-rtV 

J  0 
c  t- 

•H  VI  ■"  0 
073 

V  VI  0  VI 
O  A  >'  0 
C  U  O  3 

r-4  ft 

V)  I-  ft  A 
0  O  E  > 
l.if>  0 
3  C 
(A-H  0  0 
•H  3  1.  X 
U.  O  A  3 


TASK  EFFECTS  TASK  "STD.  DEVS."  TASK  RISKS  90;<  CONFIDENCE 

INTERVAL 


OfOOrHOOt-HOOO 

oooooooooo 

Oi  . 

lU  OOOOOOOOOC3 

o  oo  f"*  o  CO  o  so  cj 

_|  O  ro  O  rH  O  O  <~4  o  o  o 

oooooooooo 


oooooooooo 


oc 

UJ 

Cl. 

O- 

3 


rHr-O'eO'OOsor^r-^r*, 

o^r^<-(ajso«HsoiOLn<vi 

Oi^OOOOOOOO 

oooooooooo 

*As^>fviriO'omeofOio 

ooNor\Jr>.ir>f-fO'jNTf>4 

01-400000000 


r-(9>0 
OtVJO 
O  O  O 


sj-  f>«  O  rH  O  O  rW 
O  O  O  O  O  O  O 
OOOOOOO 


UJ  O  O  O 

3  WWW 

o  0-^0 


ui<<r 

0>  o  o  o 

U.UJ 

oz 


OOOOOOO 

N  I-H  O  (\J  O  K> 
OOOOOOO 
OOOOOOO 

OOOOOOO 


Ofsjso 
040  CO 
•-<*00 


oc 

lAi  O  O  O 
a.  WWW 
0.  tOtO  >3- 
3  (SIOJ  O 

r-CIOft 


»0  so  lO  GO  Os  •-< 

f"«.  <o  rs.  GO  *o  r- OS 
O  ^#-4  O  o 


OOOOOOO 


oooooooooc 


OOOOOOOOOO 


O'OOcO'JO'tncsjO'inCs 

rsJsJ-OtSIi-tOCSi— 

OOOOOOOOOO 

OOOOOOOOOO 

iorN.o'0'f'»sTfs.f>»'>3‘a' 

<SJsrO^^O.-l^-4Q 

OOOOOOOOOO 

OOOOOOOOOO 


<^4noo^o<-^<si40—^<^<y' 

(N40'0*OfMOfNj<SlfHO 

OOOOOOOOOO 


M 


< 

> 

o^a:  • 

UJ  UJ  o 
H-  2  W 

zo^ 


sO^JOs^O^s.irtC0^00NO 

>oo(MirtiA(sjtnN3^rgr/^ 

o<-<oooooooo 

oooooooooo 

<0»00'^Os•-<(SJ^JCQ^•. 

^Os^Os3fslOii^'^'3‘ 

oooooooor^o 

oooooooooo 


s0*-4v«K0fSJOU>»0*^'4C0 

0*^00000000 


<  r 
>  C 
OCQC. 
UJUJC 


oooooooooo 

UJ  o 

■U 

oooooooooo 

UJ  o 

VI 

a 

tn 

o 

*0  9 

i-i 

rv.iOrs.fv<sjfsitn^coo' 

z 

1-4 

z 

9  O  VI 

Q£ 

<sicro*onjo<M*H.-<o 

UJ 

QC 

sTrj—tDOs^ocomfsjiA 

UJ 

oooooooooo 

o 

Or-IOOOOOOOO 

a 

l.'0  4^ 

I-<  --s 

L.  C  9 

oooooooooo 

u.  «0 

oooooooooo 

U.  O 

Z  <VJ 

z  rvj 

u  an 

OQS  • 

OQi:  • 

OlJJo 

OUJO 

Vl<  V) 

O.  W 

0.  w 

•H  9 

2 

N  Q- 

2 

9 

o 

CMOOOOOOOOO 

03<M 

O 

cviooooooooo 

03fSJ 

cx  o 

o 

cginoooooooo 

cr 

a 

<M4noooooooo 

C7S 

z 

>Tsoooso<Moor^ 

o 

z 

sT'OOO'OrsjOOfs. 

o 

•H  -H 

*— • 

>-< 

4*  CTJ 

2 

2 

9*H  C 
4*  O 

oooooooooo 


oooooooooo 


oooooooooo 

>»  wwwwwwwwww 


OOOOOOOOOO 


oooooooooo 


OOOOOOOOOO 


^  fsj  m  40  m  in -sT  ir>  so  vo 


C\no40i0so‘0syios0'0 


--•(M*Os:rinsOf^oo<>o 


o<-ieoo>-<asr'.osos9. 

•-t<-<orj--<o*-<<>-<oo 

oooooooooo 

OOnU>^OsOCO"-«9scO 

^00(SI^0r-<(S}00 

oooooooooo 


oooooooooo 


4000000 


»-if«^eososro'C'jrotMOs 

•-<<-<orsj-HO(N>rsj<"40 

oooooooooo 


oooooooooo 


> 

+ 

II 


r».»-<r*»mi-<osr'.\3-soso 

tOsOGOmCTSvOCMsOOcO 

<Mtntr>iri'otOsTm'0>o 


3 

to 


cj4n4r>mso40sj‘4n'>o>o 


r-^(M»0sJ■m^^>r>.00£^O 


:>'o  a 
■  a  «  V) 

EJ)  • 
•H>H  l~ 

L  L 
O  O  O 
C  VI  O 
9 

C-D  • 

«  X 
X  VI 

O  •  VI 
VIX  c  « 
V-f*  10  VI 
4^  O  (0 
<0  E  VI  U 
E  « 

•H  I  VI 

4^  « 

VI  VIX 
«  «4^ 

3  C 
«-H  9 
L  (0  L 
9  >  9 
a 

VI  a 
9  c  c 
VIH-H 
9  w 
X  VI  VI 
4»  .h  9 
C  E  1. 

9  3 

CX  O 
94^  rl 
a-HH- 


•H  VI  •-  9 
9*^ 

4*  VI  9  VI 
O  (0  >  9 
C  U  O  3 

M  1.  a  9 

9  O  E  > 
LH-  9 
3  C 
04>  9  9 
•H  3  U  X 
U.  O  9  3 


imputed  for  missing 


TABLE  6 

SYSTEM  D  -  Missing  non-standard#  excluded 


(MOO^OO^Of-<OU~|rH 

.-imoooo^ooo 

oooooooooo 

tx  . 

lU  O  O  O  O  O  O  O  O  O  C3 

o  r*.  cs»  K) o  «-i  <\i  o  O' 

_)  •-«  O  rH  .-lO  f-4  O  O  O 

oooooooooo 


oooooooooa 


r>-sTgO'J’'OiO'OOtnvo 

•-i«ocsjr^NO*-Hr'»0‘nrsi 

*-H.-«OOOOOOUO 

0£  . 

uj  OOOOOOOOOO 

Cl.  r>.  m  f-H  *-4  so  Q  r'' tn  »o  fO 
3  oo  00  (M  m  m  r-i  in  o  O’ Csi 

0t-400000000 

oooooooooo 


o»oa«fOO'sonjo.oO' 

mmocsj^ociio^o 

oooooooooo 

oooooooooo 

«4‘or«-r'jr«-»Oso'n(Sjr*. 

niino>-<.-HO.-^o^o 

oooooooooo 

oooooooooo 


»oo'oo»HNr»oooa\a' 

^T€^o»*^cgo^no•-^o 

oooooooooo 

oooooooooo 

0'Tr«.eofOfor>.oooO' 

>TO'OCVJ<MOCSIO«-»0 

oooooooooo 

OOOOOOOOOO 

<SJOOOOOOOOO 

<sjinoooooooo 

'3'sooo'ocgoors. 


■4  (SI 


oooooooooo 


oooooooooo 


oooooooooo 


w 


oca:  • 

ujiuo 

h-3:  w 


LJ 

o 


3S  Kt 

ooc  • 

(JUJO 

0.  W 

NQ.ro 
o  3  ro 


oooooooooo 


oooooooooo 


oooooooooo 


(vjminmNom'a’in'O'o 


(sjinminvotnsT'j-'Oso 


4(sirnsTinNor-co(T'o 


or«.#-«oeoo(Nioo{v 

ofsjoooooooo 

oooooooooo 

oooooooooo 


orgoo(MO(Sio 

0/>4000000 

oooooooo 


orn 
o  o 
o  o 


oooooooooo 


'X'Oin'OOt-«in(SjsT(Si 

0'4n>-4ooo'oocoor>.(M 

O*0^O<-tOrHOO#H 

DC  . 

tu  OOOOOOOOOO 

Q.  m  o  ro  O' ro  o  rv.  m  hw  ro 

3  CM  (Ni  O' m  (Nj  CO  r*  >  >-4  s3 

•~(*00>-lr-«0#-«Or-ti>H 

OOOOOOOOOO 


srC'>Jr*>ooeoooeO'*4'Or^ 

»oomrM»n(siino(M'T 

OiF>«00000000 

OOOOOOOOOO 

(M>3“inr0O0'rHsys0'0 

•>TO'rOrN.in«-HsO»-*sJ‘'X 

OOOOOOOOOO 

OOOOOOOOOO 


•<r  *0  CM  >o  o 'O  O' o  o-«  eo  H*  2 

K> 'o  m  CM  O' rs.  o  .-4  »o  zc 

0<>400000000  >>4. 

^  oooooooooo  UJ 

1/i  WWW  WWW'./  w  U 

•-(  COOCMcO'T’r-tCJ'O^^OO  Z 

OC  'O  m  f'.  so  O  CO  O  (M  ^  Ui 

OWOOOOOOOO  o 


oooooooooo 


(MOOOOOOOOO 

Csjmoooooooo 

^sCoosorMoors. 


oooooooooo 


oooooooooo 


^  oooooooooo 


oooooooooo 


oooooooooo 


oooooooooo 


> 

+ 

2l 


Z  CM 

oor  • 

OUJO 
0."/ 
NO.  CO 
O  3  r-C 


(sjinmmvocnsTm'O'O 


(Minmm'Ou^'TinvO'o 


*-«(simsj’»n'or>.eoo'o 


3 

D 


“D  e 
«  O  M 
■H  H  e 

U  C  (S 
ro  e  E 
u  a-H 
a-f^ 
Vi<  VI 


CJZ  PI 

ov  c 

•H  -H 
-H  CD 
19  H  C 
+»  O 

3*0  a 
a  V  v) 
EJ3  4) 
•H>H  L 
L.  L 

o  u  o 

C  V)  o 
9 

CD  9 
9  £. 

X  lfl+» 

J-o 

0  9  1/) 
V)X  L.  9 

9-H  9  in 

4^  9  9 

9  E  in  u 

E  9 
•H  I  tf)  Dl 
4r  9  C 
w  inx  -H 
9  94>r  V) 
3  C  VI 
9'H  9-h 
L-  9  L  E 
9  >  9 

a  C 
V)  cn  o 
9  C  C4- 
Hhh 
9  M  T} 
X  V)  in  9 
4^  H  9-H 
C  E  1.  3 
9  3  a 

LX  01  E 
9-Hrtr! 
a-HH- 

t 

•rt  in  -  9 
973 

-H  in  9  in 

O  9  >  9 
C  U  O  3 

V)  L  a  9 
9  O  E  > 
L  H-  9 
3  C 
Ol-H  9  9 
•H  3  LX 
IL  O  9  3 


TASK  EFFECTS  TASK  "STD.  DEVS."  TASK  RISKS  905:  CONFIDENCE 

INTERVAL 


2 

O 


r«-r-sc4i-Hr'.fsjm(MNy 

OOOi-H(SJOi^O.-4 

ooooooooo 

syo>«ooor>-'TsTf^O' 

0000<-«000«-4 

ooooooooo 


(vjK)o."H»oo'»Hvrm 

<^J^v^Teooa'^JO^«- 

(MO>-l(Si\00>3'rHN^ 

oc  . 

lU  ooooooooo 
Cl.  WWWK^WWWWK^ 
d.  NT  ■<»  »H  O' nJ- o  f-«  vj- CM 

3  oo  ON  CO  »o  r>. 'T  oo  r>- 

^OOrHCJOrHOCM 


to  ^  oo 

tAOCsJ 

o  o  o 

a:  .  .  . 

UJ  ooo 

3  ww^ 

Oi  O  (MCnIOn 
O  CAOCVJ 

z  — 1  ooo 

Q  >  ooo 

i-ta: 

U.UJ 
Zl~ 

oz 


'O- fvj  nt  NT  o  in 
m  in  r-«  o  rs. 
(sjorNiOfH 

o  o  o  o  o  o 

sOiAmeOCMON 

eo  o  o  csj  .-to 

O  i-t  O  .-t  O  1"^ 

o  o  o  o  oo 


CONOf^ 

O  .-t 


O  CO  o  O  nT  *-t 
O  On  sT  *0 
m  in  (\j  m  rH  nt 


UJ  ooo 

a.  WWW 
o.  »noN»n 


o  o  o  o  o  o 
cocsiomNom 

nT  On  1/)  ^  I/) 

^  nT  «-*  nT  -t  O' 


ooooooooo 


ooooooooo 


01 

o 


lU 

o 

o 

z: 


mocONTCNJeONTNOfO 

ocgromofMcoCNjfN- 

OOOOf-tOOOO 

ooooooooo 

in(\joinNTin»ncNNT 

»0.H<M»0»n.HCNJ^NT 

ooooooooo 

ooooooooo 


CnJ.HnT*-<0<M'TnO»0 

'Ts^^N-o^^lncoln^T 

fHOOf-i^OCMOCNJ 

^  OOOOOOOOO 
</>  wwwwwwwww 
*-t  CO  NT  ^  nt  m  ON  >T  m  <M 
cjc  .-H  to  m  r«.  *n  o  NT  O' 

^0O*"4«-lO^OrH 

OOOOOOOOO 


2 

o 


OOOOOOOOO 

OOnOOCNICsJOOOO 

x-tCM  oHOCNJcOcO 


>  <M 
OelOi  • 
UJUJO 

zoo 

-H  ^Cn4 

UJ  o 

J 

z 

UJ 

o 


Z  to 
OQC  • 
U  UJ  o 

Q.  W 

•  TO.  to 

OOnT 


ooooooooo 


OOOOOOOOO 


ooooooooo 


OOOOOOOOO 


ooooooooo 


OOOOOOOOO 


^  to  T"  nT  fsj  nT  NO  r«- m  f". 

p-  WWWWWWWWW 


to^osTCNj-'T'or^iOr'. 


l/> 


-tiomNor^coo.- 


ro.NTNor^'-tNAh^oto 

fN.*oinaNoinoNNTco 

QC 

OOOOt-tOOOO 

UJ 

ooooooooo 

C) 

eOfUNTTNlONf-tCSJcOm 

r^sTNOOp-tmoNTON 

tn 

to 

OOOi-4.-tOi-tOO 

tn 

ooooooooo 

_j 

< 

to 

> 

."4 

ti£.0C 

^  ^  ^  ^  ^ 

UJUJO 

< 

On»O.HOnO^COOO^C 

»-2 

w 

COvT'^CONO(MnN«0 

ZOCSI 

^Oi-l<SJ^OtOOK» 

IH  _l 

O 

ooooooooo 

UJ 

O 

w 

to 

wwwwwwwww 

o 

-p  9 

tni-t^cooinsor""^ 

Z 

9  U  M 

Or: 

eou^fgtOFHNOfveorN. 

UJ 

■H'H  9 

r"lO^(SJtOO<MO(SI 

o 

U'D-t"' 

M 

L  C  ID 

ooooooooo 

li. 

NO 

ID  «  E 

z 

to 

O  a-H 

ou: 

ouio 

W<  (A 

a. 

rt  9 

2 

^:Q_ln 

9 

O 

ooooooooo 

o  ^  tn 

cx  o 

Q 

coNOorucsioooo 

ON 

o-t^  c 

Z 

.-4CNJ  i^ocsjeoco 

o 

•H  'H 

i-t 

r^ttsi  •-! 

C*D 

2 

ID'H  C 

't't  o 
D"o  a 
a  9  M 

tn<siCN.-»r"-<-^rN.toct) 

EX  9 

c 

0£ 

.-t>-*Or-«Oi-tOr-tO 

■H'H  U 

UJ 

L.  L 

to 

ooooooooo 

O  U  O 

> 

. 

wwwwwwwww 

C  W  O 

UJ 

o 

nOnT^OoO>-^On>o9n 

9 

a 

c- 

r-l.-t.-t^O^Op-tO 

CT?  9 

to 

ooooooooo 

9  X 
X  VI 

o 

♦- 

O  9 

VI 

to 

^  ^  ^  *  .“N  ^  ^ 

VIX  L. 

9 

e 

NTNO>^NOcotmnoNT 

V'**  m 

VI 

commintoinrucom 

4^  9 

9 

9  E  V) 

O 

to 

ooooooooo 

rv 

E  9 

< 

wwwwwwwww 

Of 

o 

•H  t  M 

CD 

.-tON^csjoruoN'T'i-t 

UJ 

4^  9 

C 

O'N^'^inNTintoa'tn 

o 

VI  tfIX 

•H 

w 

9  94^  n 

ooooooooo 

a 

NT 

3  C  M 

rH 

9^9 

•H 

tn 

U  ID  L 

E 

o 

ID  >  ID 

CJ^»ONO<-HNTONOtO 

a  1. 

o: 

i.*Hp-iOC'J»-< 

VI  01 

O 

UJ 

9  C  CH- 

OOOOOOOOO 

VI  ‘H  -H 

9  VI 

*n 

to 

o 

K>rN.rN.inruNTi""(roto 

X  (A  VI 

9 

h- 

4'  H  9  4'' 

u 

to 

/■N 

C  E  1. 

3 

UJ 

ooooooooo 

9  3  a 

u. 

rj 

i.X  CD  E 

u. 

«4^>H 

•H 

o 

anM- 

o.-tCT'inNTruvofNjco 

w 

3 

9 

NOOinNor-.-t»nNT«-« 

o 

m 

c 

L 

to 

o 

•H  VI  -N 

9 

►O'T^fNjNT'orN'inr^ 

9*0 

1— 

wwwwwwwww 

o 

4^  VI  9 

VI 

NO'TOtOON<N4NO,-,00 

O  ID  > 

9 

zl 

in^inNoini-tNTNTf-t 

COO 

3 

totoNTfsiNT'Or-mr*. 

VI  L  a 

9 

9  O  E 
UH-  9 

> 

v: 

3 

C 

to 

04^  9 

9 

< 

•-ctoin'or'*cooi“»rsj 

•H  3  L.  X 

h- 

^  ^  ^ 

U.  O  ID  3 

TABLE  8  , 

SYSTEM  L  -  Missing  non-standard,  excluded 
-  Tasks  2, A, 9  excluded 


^eoor>.r^(NJC}r«.o 

OOOi-«C\IOt-«Q.--* 

ooooooooo 

fOOCSJCJsO^OOO'JT^ 

OOOOi-HOOOr^ 

ooooooooo 


eoiooo«~tr^o(^<o^ 

MOiotniOr-ir^ONO 

OOOr-4(SJO(\JO<-( 

OC  . 

tU  OOOOOOOOO 

C3  fsj  >£>  ir>  r«.  rv  00  CM  o 
«  fsj  o  »0  O' rH  O  to  >0 
0000>-^0W0rW 

OOOOOOOOO 


ro»-<oor'-'Ot*><-»or~ 

«j'ooior^oooeo»-^iO 

,-HO^fNJ‘0^'y'-i»0 

OOOOOOOOO 

coCT'tO'j-Of'.iOrN.oO 

OOOfMfMOe^OrJ 

ooooooooo 


toco«J*otooNoo»Or*>. 

nTp^O'OO'O'CO'J’*-* 

ooooooooo 

lAeO'OtOv^^^-Hino’O' 

ooooooooo 


O'OorjiooO'O'oo 

•  cj  <M  <r  lO  o  fM  oO  Cm  r«- 

0£.  O  O  O  O  «-•  O  o  o  o 

UJ  . 

ooooooooo 

Q  nJ"  O' 00  nT  '3‘ rH  oo 

KT)  ooooooooo 
ooooooooo 


lf\tf\tOOOO'3’(M»-lr-< 

•  sohomo'Oino'Too 
o'  o  o  O  O  r-«  O  f-*0  o 

UJ  . .  -  . 

ooooooooo 

A  \y  4^  (M  <M  o  .-H  o  f'* 

^  (s^  m  fN.  o  *-<  .-I  O' 

^/^  000r-<e-^0^00 

ooooooooo 


OOOe-*'TO»0O<M 

ooooooooo 

rs.in'TO'f^flOtor- 

■.^KiinNyr«-»'^o>Teo 

aoo.-<^o^Qe-i 

ooooooooo 


^iA»-«O''T0'r-tr'*0' 

w0rHCM'3'0'r0<M 

ooooooooo 

^rVxJ-O^N^eO'^O 

fgintMNj^O'Or-cor'. 

^OfHCM'OOCMOCM 

OOOOOOOOO 


OOOOOOOoO 
oo^oo<\l^Joooo 
,-4  (SI  •-»  O  CM  go  oO 
-ACM  e-t 


OOOOOOOOO 
qO'OOCmcmoooo 
,-((M  .-<o(Meo<0 


ooooooooo 

Q  fs.  M  sT  rg  ^o  <-f  O' ^ 

^  I-H  rH  O  i—t  r-t  eH  rH 

. 

ooooooooo 


•-irHO(M>-<inOOsOO 

sovoio^otO'^CMr'.sr 

ooooooooo 

r*.  *0  CM  'O  eH  iTI 

ooooooooo 


cor^r'O'eHfOorgf'J 

rHrH^O^^'-*CV)r-l 


OOOOOOOOO 


C00^0^^-^c^^^p^ 

<MfMrg«-*<'JCM*-«(sieH 


OOOOOOOOO 


CMoO'^kOO'i-HCMmco 

'OO'or'-r'-CM'OuiCM 

»ONT>.T<'g'y'or''4nr'- 

.-if'va'tooO'-Hirv'T 

K>gotO'0'^OtO»Of-< 


rHfOCfTsor'OOOf-HCM 


•  \x<vo0'-*'00'^»i'r«- 

QL  O  r-<  O  I-H  O  •-<  O 

Ui  . 

ooooooooo 


OOOOOOOOO 


t-*  OO  'O  ^  O  (M  o 

eoi^^tnu^tompHgou^ 


OOOOOOOOO 


vOC7'«C<MC7'(M'^> '**>•-« 
*0M?'O<in^^tncMO*«n 


ooooooooo 


(SJ.-4i-<»-«rHrHO(M»-C 


ooooooooo 


»Ofs.r^ir)CM'JT^^'>»o 

(M,_(r-lr-«r-4>-<OCMi-H 


ooooooooo 


rs.  p-i  O' lO  CM  »o  CM  to  3 
o  tn  so  "O O 

01C10»00'CM'J'»-^<0 

tOtO«jrM'3’'Or-'iOr- 


,-t»0‘ri\or'«*oo» 


CTJ  ® 

«  X 
X  W  4^ 
J-O 
O  «  M 
M  X  U  V 
®4^  (9  M 
4^  V  Q 
(Q  E  lA  U 
E  « 

•H  I  lA 
•4*  9 

4A  WX 

« 

3  C 
9^  9 
L  10  C 
(0  >  (0 
a 

(A  CA 

®  c  c 

lA-HH 
9  W 
X  iA  (A 
•f'  H  ® 
CEL 
9  3 

LX  O 
lOV-H 

a-rCH- 

3  9 

C  L 
•H  (A  (0 
9~0 

•H  lA  ®  (A 
O  (0  >  « 
C  U  O  3 

lA  L  a  <0 
®  O  E  > 
C  >4-  ® 

3  C 
Ot+'  9  9 
•H  3  LX 
U.  O  10  3 


imputed  for  missing 


Figure  1 .  Uniform  Residuals 
System  L,  Model;  Log  N.  no  imputation 


Figure  2.  Uniform  Residuals 
System  L,  Model:  Log  N,  imputation 


Figure  3.  Uniform  Residuals 
System  L,  Model;  Log  EV.  no  imputation 


Figure  4.  Uniform  Residuals 
System  L.  Model:  Log  EV,  imputation 
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Figure  5.  Uniforrr  Residuals 
System  D,  Model:  Log  N,  no  imputation 
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Figure  6.  Uniform  Residuals 
System  D,  Model;  Log  N,  imputation 
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Figure  7.  Uniform  Residuals 

System  D.  Model:  Log  EV,  no  imputation 


Figure  8.  Uniform  Residuals 
System  D,  Model:  Log  EV,  imputation 
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